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Abstract 

In some models with an extra U{1) gauge boson Z', the gauge coupUngs of the Z' to different 
generations of fermions may not be universal. Flavor mixing in general can be induced at the tree 



, level in the up-type and/or down-type quark sector after diagonalizing their mass matrices. In this 

^) ' work, we concentrate on the flavor mixing in the up-type quark sector. We deduce a constraint from 



mixing. We study in detail single top-quark production via flavor-changing Z' exchange 



Qh. at the LHC and the ILC. We found that for a typical value of Mz' = 1 TeV, the production cross 



section at the LHC can be of the order of 1 fb. However, the background from the Standard Model 
single top-quark production makes it difficult to detect the flavor-changing Z' signal unless with 
a decent charm tagging method. On the other hand, at the ILC, the production cross section at 
the resonance energy of ^/s ~ Mz' can reach a size of more than 100 fb. Even away from the 
resonance, the cross section at ILC is shown to be larger than the threshold of observability of 0.01 
fb. 
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I. INTRODUCTION 



Searches for flavor- changing neutral currents (FCNC) have been pursued for many years. 
So far the sizes of FCNC in the u-c, b-s, s-d, and b-d sectors are in general agreement 
with the Standard Model (SM) predictions, namely, those given by the Cabibbo-Kobayashi- 
Maskawa (CKM) mechanism. In the SM, tree-level FCNC is absent in both gauge and 
Yukawa interactions. They can only arise from loop diagrams, such as penguin and box 
diagrams, and are therefore highly suppressed. Nevertheless, one-loop FCNC processes can 
be enhanced by orders of magnitude in some cases due to the presence of new physics, see 
Ref. for a review. Tree-level FCNCs via some exotic gauge bosons are empirically allowed 
only if these bosons are sufficiently heavy or their couplings to SMparticles are sufficiently 
small; otherwise, they would have been ruled out by current data 0,0,0,0,0. 

However, the effects of FCNC involving the top-quark are not yet well probed experi- 
mentally, at least not by the present data. From the existing LEP and Tevatron data we 
have only very weak constraints on the t-q-Z and t-q-'j FCNC couplings. These constraints 
will not be improved any further until the operation of the Large Hadron Collider (LHC) 
or perhaps a future International Linear Collider (ILC) is built. The goal of the paper is 
to analyze effects of tree-level FCNC interactions induced by an additional Z' boson on the 
up-type quark sector in general. In particular, we study the tc + ic production at the LHC 
and ILC. 

Examples of Z' arising from some grand unified theory (GUT) models are 0: 

Z^ occurring in Eq — > S'O(IO) x f/(l)^ , 
Z^ occurring in SO (10) SU{5) x U{1)^ , 



Zrj = COS Z^ — sin 9 Z^ , cos 6 = -y/s/S . 



In these examples, the SM fermions together with an additional right-handed neutrino are 
placed in the 16 of S'O(IO) embedded in the 27 of Eq. One expects in such models that the 
Z' boson will couple universally to the three generations of fermions and thus the couplings 
are diagonal in the flavor space. ^ However, it is possible that exotic quarks like h and h'^ in 



^ Since the vector- and axial-vector-current interactions of Z' always couple to either two left-handed fields 
or two right-handed fields, the unitary rotations of the gauge eigenstates to the mass eigenstates will 
always preserve the diagonality of the Z' interactions if the chiral couplings are family-universal. 



2 



the 27 of i^e may have their f/(l)' charges different from the left-handed and right-handed 
down-type quarks. In this case, the SM quarks will mix, leading to in general both Z- 
and Z'-mediated FCNCs. We note that flavor- changing Z' boson can also arise in certain 
dynamical symmetry breaking models 

In some string models, the three generations of SM fermions are constructed differently, 
resulting in family non-universal Z' couplings to fermions in different generations. As a 
first step, we consider the particular case that the Z' couples with a different strength to 
the third generation, as motivated by a particular class of string models jsl. Once we do a 
unitary rotation from the interaction basis to mass eigenbasis, tree-level FCNCs are induced 
naturally. Several works have been done regarding the FCNCs in the down-type quark sector 
recently J, 0,1^. The same can occur in the up-type quark sector too. In order to increase 
the predictive power of our model, we assume in this paper that the left-handed down-type 
sector is already in diagonal form, such that Vckm = KIl) where V^i^ is the left-handed up- 
type sector unitary rotation matrix. Since we do not have much information about both the 
right-handed up-type and down-type sectors, we simply assume that their Z' interactions are 
family-universal and flavor-diagonal in the interaction basis. In this case, unitary rotations 
keep the right-handed couplings flavor-diagonal. Therefore, the only FCNCs arise in the left- 
handed t-c-u sector and depend on the CKM matrix elements and one additional parameter 
x, which denotes the strength of the Z' coupling to the third generation relative to the flrst 
two generations. Consequently, if x is an 0(1) parameter but not exactly equal to 1, the 
t-c-Z' will produce the largest FCNC effect. 

Associated top-charm production at the LHC or ILC in the SM is expected to be very 
suppressed [lO| . However, the rates enhanced by the presence of new physics such as SUSY, 

may reach observable rates in 
some cases, and can then be used to probe FCNC couplings. Single top production can 
also proceed through the introduction of anomalous couphngs: t-q-g^ t-q-'y and t-q-Z [3] at 
both hadron and e^e~ colliders. Such model independent analysis are useful in probing the 
strength of observable FCNC couplings. Many detailed studies of the Z' phenomenology 



have been done in recent years 



oupiings. Many aetaiiea stuaies c 
Q,3,IIfly,|l3, 14, 15, 16,|l7,Q 



In this work, we study the capability of the LHC and the ILC to identify the t-c FCNC 
effect by measuring the production of tc + tc pairs. Since most of the cross section comes 
from the s-channel production of the Z', these types of FCNC processes will be searched 
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only after the Z' is discovered. The most obvious channel to discover the Z' is the Drell-Yan 
process at the LHC, in which a clean resonance peak can be identified in the invariant mass 
spectrum Mijt^i^^ of the lepton-antilepton pair. Experimenters can then search for the 
hadronic modes with an invariant mass reconstructed at the Z' mass. Those involving the 
top-quark may be somewhat complicated because of the 3-jet or 1-jet-l-lepton- decay 
products of the parent top-quark. But in principle they can be measured, though at lower 
efficiencies. At the LHC, however, the SM single top-quark production presents a challenging 
background to tc + tc production. Unless one can efficiently distinguish the charm-quark 
from the bottom-quark and the other light quarks, the SM single top-quark background 
makes the FCNC tc + tc process very pessimistic. There may be a slight possibility of D- 
tagging but it is still too early to tell its efficiency. On the other hand, an e"^e~ collider or 
the ILC is an ideal place to search for tc + tc FCNC production. One can measure the ratio 
of the production rates for ti : tc + ic : cc to identify the FCNC in t-c sector. Also, charm 
tagging is considerably easier in the e^e~ environment. At any rate, one can simply measure 
the ti pairs and a single top-quark plus one jet (either c or u) in the hadronic decays of the 
Z' boson. We will estimate the potential of this approach in this paper. 

The organization of the paper is as follows. In the next section, we outline the formal- 
ism of the model. In Sec. Ill, we derive the current limit on the c-u transition from the 
D^-D^ mixing. We calculate the production rates of various channels and estimate their 
detectabilities in Sec. IV. Our conclusion is presented in Sec. V. 



II. FORMALISM 

We follow closely the formalism in Ref. ^. In the gauge eigenstate basis, the neutral 
current Lagrangian can be written as 

£nc = -eJlA, - g.j'^'^^Zl - g^J^'^^Z^ , (1) 

where Z° is the SU{2) x f/(l) neutral gauge boson, Z2 the new gauge boson associated with 
an additional Abelian gauge symmetry. We assume for simplicity that there is no mixing 
between Zi and Z2, then they are also the mass eigenstates Z and Z' respectively. The 
current associated with the additional f/(l)' gauge symmetry is 

.(2) p 



J2) p 



(2) 



where e^"* is the chiral couphng of Z2 with fermions i and j running over all quarks and 
leptons. If the Z2 couplings are diagonal but family-nonuniversal, flavor changing couplings 
are induced by fermion mixing. 

n 

Z'- mediated FCNCs have been studied in detail in Ref . jj] for the down-type quark sector 
and their implications in B meson decays. Since such an effect may occur to the up-type 
quarks as well, we concentrate on this sector in this paper. For simplicity, we assume that the 
Z' couplings to the leptons and down-type quarks are flavor-diagonal and family- universal: 

R — Qlr^^ ^lr — Qlr^ — Ql^ where 1 is the 3x3 identity matrix in the 

generation space and ^, Qj^ ^ and are the chiral charges. On the other hand, the 
interaction Lagrangian of Z' with the up-type quarks is given by 



(2) 
NC 



(3) 



where the subscript / denotes the interaction basis. For definiteness in our predictions, we 
assume 



/ 



el 



Ql 



1 



1 

\0 a;y 



and 



~R 



Ql 



1 



1 

Vo I) 



(4) 



That is, only the left-handed couplings are family non- universal. The deviation from family 
universality and thus the magnitude of FCNC are characterized by the parameter x in the 
Il-Il-Z' entry, which we take to be of 0(1) but not equal to 1. Ql ^ are the chiral f/(l)' 
charges of the up-type quarks. The chiral charges need to be specified by the Z' model of 
interest . 

When diagonalizing the up-type Yukawa coupling or the mass matrix, we rotate the left- 
handed and right-handed fields by Vul and Vur, respectively. Therefore, the Lagrangian 



£nc becomes 



C 



(2) 
NC 



(5) 



c 

V ^ / 

where the subscript M denotes the mass eigenbasis. With the form of e\ assumed in Eq.(j3]), 
the right-handed sector is still flavor-diagonal in the mass eigenbasis, because is propor- 



tional to the identity matrix. However, V^i^e^VuL is in general non-diagonal. With the fact 
that VcKM = V^iVdL and our assumption of the down-quark sector has no mixing, 

The flavor mixing in the left-handed fields is in this case simply related to Vckm, making 
the model more predictive. Explicitly, 



Ql 



(6) 



/ 1 {x-i)VubV:, {x-i)v^bV;,\ 

{x - m^v:, 1 (x - m>,v,i 
\{x-m,v:, {x-m,v:, x j 

where we have used the unitarity conditions of Vckm- It is easy to see that the sizes of the 
flavor-changing couplings satisfy in the following hierarchy: \B'^^\ > |-B£"| > \Bff'\. Note that 
the right-handed couplings are flavor- diagonal and are of 0(1). 

The following Z' models will be considered in this work: (i) Z' of the sequential Z model, 
(ii) Zlji of the left-right symmetric model, (iii) Z^ occurring in 5*0(10) — > SU{5) x f/(l). 



(iv) Z^ occurring in Eq 50(10) x f/(l), (v) Z^ = y3/8Z-^ — y5/8Z^ occurring in many 
superstring-inspired models in which Eq breaks directly down to a rank-5 group [1^. In the 
sequential Z model, the gauge coupling g2 = gi and the chiral couplings are the same as the 
SM Z boson. In the other models, the gauge coupling takes on the value 

92 = sin^^^i 

where is 0(1) in string-inspired models and 6„ is the Weinberg angle. We simply choose 
Xg = 1 throughout. The chiral couplings of the Zlr in the left-right symmetric model is 
given by [l^ 

Ql = -\^(^) (B-L),, (7) 



, 5 V2a 

Qr = Vf («^3i^-^(^-^)^) ' (8) 
where B and L denote the baryon and lepton numbers of the fermion i, respectively. T^r 
is the third component of its right-handed isospin in the S'f/(2)R group. In the left-right 
symmetric model with ql = qr, the parameter a is given by 

/1 o _:„2 n \ 1/2 



where we have used sin^ 5^ = 0.2316. The chiral charges for these various Z' models are 
compiled in Table HI 

Before ending this section, we quote current limits on an extra f/(l) gauge boson from 
direct searches at colliders. The most stringent limits are given by the preliminary results 
from CDF j2Q| at the Tevatron: 

> 845 GeV , 
Z^ > 720 GeV, 
Z^ > 690 GeV , and 
Z^ > 715 GeV. 

In the following, we will use a typical value of Mz' = 1 TeV unless otherwise stated. 



III. CONSTRAINTS FROM D^-D^ MIXING 
A. D^-'W mixing in SM 

To second order in perturbation, the off-diagonal elements in the neutral D meson mass 
matrix contain two contributions from short-distance phy sics. One part involves |AC| = 2 
local operators from box and dipenguin diagrams |2ll. I22I at the niD scale, contributing to 
only the dispersive part of the mass matrix. Due to a severe CKM suppression in the SM, 
the contribution from this part is negligible. The other part involves the insertion of two 



TABLE I: Chiral couplings of various Z' models. 



Sequential Z Ziji Z^ Z^ Z^ 

-1 1 -2 

1-12 



R ' ' 2VT0 2VT5 

11-2 



Ql 0.3456 -0.08493 

-0.1544 0.5038 

Qi -0.4228 -0.08493 5^ ^ 

0-0772 -0.6736 ^ ^ 

-0-2684 0.2548 iTTo ^ 



1-12 



Qn 0-2316 -0.3339 i7T3 731 iTTI 

Ql 0.5 0.2548 



3 11 



2V10 V24 2V15 
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I AC I = 1 transitions, contributing to both the dispersive and absorptive parts of the mass 
matrix. 

Since CP is a good approximate symmetry in D decays, we have the CP eigenstates \D±) 
with CV\D±) = ±\D±) as the mass eigenstates too. It is convenient to define 

AmD=m+-m_, AF^ = r+ - F. , rB = l(r+ + F_), (9) 

and consider the dimensionless parameters 

xd = — and yn = ■ (10) 

The short-distance contributions to x^i and yn have been evaluated to the next-to-leading 
order (NLO) and both found to be about 6 x 10~^, quoting the central values from Ref. [2^. 



They are far below the current experimental constraints. In contrast, the 
effects are expected to be more dominant but difficult to estimate accurately 



ong-distance 



24|. 



B. D^-D^ mixing in Z' models 

As shown in Sec. II, in Z' models one can generate off-diagonal Z' coupling to charm 
and up quarks. Due to the large Z' mass, this can induce tree-level processes for the D^-D^ 
mixing. Therefore, the |AC| = 2 operators receive new contributions. However, it has less 
influence on the long-distance physics. In view of the smallness of the SM contributions 
through the double insertion of |AC| = 1 operators, here we want to estimate the pure Z' 
effect on checking whether our model contradicts with current experimental bounds on 
D^-Jjo mixing. 

At the scale, the most general |AC| = 2 effective Hamiltonian due to the FCNC Z' 
interactions is: 

n^s = ^ [uYiCrPL + CI^Pr)c] [ui,{CTPl + CI^Pr)c] + h.c. , (11) 

where Cf^^j^ are generic left- and right-handed Z' coupling to u and c quarks. Since we 
suppose there is no flavor- changing couplings for the right-handed fermions, C"^ = and we 
obtain: 

8 



where gi = e/{swcw) and O = [wy^{l — 75)c][m7^(1 — 75)0]. Therefore, its contribution to 
the neutral D meson mass difference is 

Am|' = 2IM12I = 2-^ < D'^in^slD^ > 

where {D^\0\D^) = ^m'j)fl)BD has been used. Numerically, we obtain 

Amf; ^ 3 X 10-5. (^^^) ' (Crf GeV , (14) 

where we take (72 = gi and the D meson decay constant J'd = 300 MeV. In the vacuum 
insertion approximation, the bag parameter Bd = 1. This is translated into 

x|'^2xlO^(C„^,)' . (15) 

Note that C^^ = Ql{x - l)KfeV2, — 1-5 x 10~'^(x - l)Ql, where we have neglected the 
renormalization group running effects in comparison with the uncertainties in the Q"^ and 
the X parameter in the model. Therefore, x£ ^ 4.6 X 10-^(a; - 1)2 (g^)^ 

The current limits from the Dalitz plot analysis of Dq — > K^~^tt^ by CLEO are (—4.5 < 
xd < 9.3)% and (-6.4 < yo < 3.6)% at the 95% C.L. [sj. Assuming negligible CP 
violation in the system, a recent Belle analysis using the —>■ K^n^ decays from 400 
fb~^ of data yields to obtain x% < 0.72 x 10"^ and -9.9 x 10"^ < v'd < 6-8 x 10"^ at 
95% C.L. Q- 

Here the modified dimensionless parameters x'j^ = xdcos6ktt + Ud sin 6kit 
and y'^ = yncosSKn — XDsm6Kn, with Sk-k being the strong phase difference between the 
doubly- Cabibbo-suppressed and Cabibbo-favored amplitudes. It is easy to see that as long 
as the combination {x — 1)Q^ is less than about 0{1), the experimental bounds can be well 
satisfied in the various Z' models that we have mentioned in the previous Section. 



IV. COLLIDER PHENOMENOLOGY 
A. Decay width of Z' 

We include only the fermion modes in the computation of the Z' decay width. The 
partial decay width of Z' W^W~ is suppressed by the Z-Z' mixing angle which is 



9 



100 




1 

0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 

M^' (TeV) 

FIG. 1: Total decay width of the Z' boson in various Z' models. 

severely constrained by electroweak precision data Therefore, Z' W~^W~ is not 

included in the total width. The general formula for the partial width into //' is given by 

4871 



/i2 



+(1 + /ii - /i2)(l - /ii + /i2) + 12v/a^ 7^e [Qi^ Q 



(16) 



where Nf = 3(1) for quark (lepton) and A(l,yUi,/i2) = (1 — yUi — /^2)^ — 4/xi/i2 with /ii = 
m'j/M'^, and /i2 = m'j,/M'^,. We include only the flavor diagonal modes e~e+, fi^fi^, t^t^, 
i^ei^e, i^ni^fi, i^tI^t, uu, dd, cc, ss, ti, and bb in the calculation. Thus we have Q^Jj^ = QlrS^^\ 

with Qlji = Ql,R = QL,Ry Q'l = Ql = Q'l = Ql 5 Qi,_R = Ql,/? = Ql,/? = Ql,/? 

Q\ ^ = The numerical values of these chiral couplings can be obtained from Table HI 

We show the total decay width of the Z' boson versus Mz' in Fig. ^ As it can be seen, 
the total width of Z' is a few to a few tens of GeV in all Z' models that we list in Table. IB 
It turns out that the branching fractions of all fermionic modes are not sensitive to the Z' 
mass. For instance, the branching fractions of Z' qq, Z' uu, Z' and Z' — > ti 

decays are in percentage of 68, 20, 10 and 2, respectively, for the sequential Z model. As we 
will see later, even the largest flavor non-diagonal mode tc + tc has a tiny branching fraction 
of 10"'^. We will take a typical value of Mz' = 1 TeV in subsequent analyses. The decay 
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FIG. 2: Contributing Feynman diagrams for qq annihilation into tc via the Z' boson, 
widths of such a Z' in various models are give in Table |n] 

B. Hadronic production of tc + tc 

Let us define our notation for the convenience of the following formulas. The momenta 
of the incoming quark and anti-quark, outgoing top and outgoing anti-charm quarks are 
denoted by pi, p2, ki and k2, respectively. We neglect the quark masses of the incoming 
partons. The Mandelstam variables are defined as follows 



s = 


(pl 


+ P2f={kl + k2f 


i = 


{Pl 


- klf = {P2 - k2f = 


u = 


{Pl 


- k2f = {P2 - k,f = 


Uc = 


u - 


■ ml , Ut = it — ml 


ic = 


i- 


ml , it = i-ml , 


Sz' = 


s — 


Ml, +iTz'Mz' , 



mf + ml s 



2 2 

ml + ml s 



'l-f3cose*) 



- -(l + /5cosr) 



TABLE II: Total decay widths of a Z' of Mz' = 1 TeV in various models. 



Model 


Sequential Z 


Zlr 




z^ 




Fz' (GeV) 


27 


20 


11 


4.9 


5.7 
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where = A^/^(l, m^/s, m^/s) and 6* is the scattering angle in the center-of-mass frame of 
the partons. The imaginary part in the Sz' is the Breit-Wigner prescription for regulating 
the Z' pole. 

At hadron colliders, the production proceeds via the conventional Drell-Yan s-channel 
mechanism as well as the t-channel diagram (Fig. |2I). The s-channel diagram dominates 
when is close to Mz'- Suppose we write the amplitude M. = M.s + M-ti after summing 
over final-state helicities and colors and averaged over initial-state helicities and colors, the 
amplitude squared is given by 



(17) 



where 



4 a 



2(|Q1P + |Q 



q |2 
r\ 



-2 {\Ql\' - \Q%\') (iQif - |g*,f ) {-uA + U) 
+8m,mt {\Ql\' + Tie {Q^Qt) s , 



(18) 



4 a, 



2 (iQi^r + \Qt?) {m? + m?) (^c^. + ks -mi- mi) 



mj.mf 2m^m? ^ 



2Mi 



Ml 



-2 (igi^r - \Qt?) {m? - m?) {-ucut 



m„ 



and 



(19) 



1 



at 



3 2sz'tz' 



27^e [Q'lQTQlQT + QtQTQlQt) ( 2«c«* + 



Ml, 



+2m,m, 7^e {Q'IQTQ'rQT + Q«QrQiQf ) (25 + 



mi; 



(20) 



The interference term needs to be included for the subprocess cc tc only. Thus, it 
simplifies down to 

1 



9l 



3 2sz'tz' 

-^2m,m,7^e (gl^g^^^lQ^r + IQll') (25 



M|, 
M2,] 



(21) 
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FIG. 3: Production cross sections for pp tc + tc at the LHC versus the Z' mass. Results for the 
five models mentioned in the text are presented. 

In the above equations, the chiral charges Q'f, Q^l and Q'l^ are given by the off-diagonal 
matrix elements of the matrix B"^ defined in Eq. ®. For instance, = (x — l)Q'%VcbV^ 
etc. Our simplified assumption made in Sec. II implies that all the off-diagonal right-handed 
chiral couplings vanish. The partonic differential cross section in the parton rest frame is 
given by 



da 



. ,. -, . (22) 

dcos^* 327rs^' ' ^ ^ 

The partonic cross section is then convoluted with the parton distribution functions, for 
which the leading order fit (L) of the CTEQ6 sets j^] are used. We show the production 
cross section of tc + tc at the LHC versus the Z' mass in Fig. El The major portion of the 
cross section comes from the kinematic region where is close to the Z' mass. We present 
our results only for tc + ic production, it is clear from Eq. (jH)) that the production rate for 
tu + iu is relatively suppressed by iKib/KbP- 
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C. Detection of t and c and backgrounds 



As we have mentioned in the Introduction, the associated top-charm decay mode of the 
Z' can in principle be discovered via the Drell-Yan channel. After knowing the mass of the 
Z' to some precision, one can look at the hadronic decays of the Z'. Among the hadronic 
decays containing the top-quark, we determine if it contains one or two top-quarks. If there 
are two top-quarks, it may be just the flavor-diagonal decay of the Z'. However, if there are 
only one top plus an another heavy-flavor jet (the c quark) in the hadronic decays of the Z', 
we identify it as the FCNC signal that we are searching for. 

The most serious irreducible background is the SM single top-quark production. We 
calculate the SM single top-quark production cross section using MADGRAPH 2^. The 
single top-quark production receives contributions from the following subprocesses 

qq ^W* ^tb + ib 
qg tbj + ibj 
bg tjj 

where j denotes a light quark jet. It is mainly the b quark in the final state that may be 
mis-identified as the charmed jet of the signal. Both the charmed and bottom jets can be 
identified using the secondary vertex method, and both of them can give rise to a displaced 
vertex in the silicon vertex detector. That is why the SM single top-quark production is the 
most serious irreducible background in this FCNC signal search. One can, however, use the 
secondary vertex mass to further distinguish between the charmed and bottom jets, as we 
shall explain in the next subsection. Before we come to that, we use some kinematic cuts to 
reduce the background cross section to about the same level as the signal cross section j^]. 

One obvious cut to reduce this background is to require a very high transverse momentum 
for the top-quark or the heavy- flavor jet, because of the heavy Z'. We employ 

PT{t),PT{j) > 350 GeV, \y{t)\,\y{j)\< 2.5 (23) 

for Mz' = 1 TeV. The rapidity cut \y{j)\ < 2.5 is due to the coverage of the central vertex 
detector. The hadronic calorimetry can, however, go very forward and backward up to about 
y = 4.5 or 5. Since there is an additional jet in the subprocess qg tbj, we can employ a 
jet veto to eliminate the events with the third jet defined by 

PtU) > 15 GeV , \y{j)\ < 4.5 (Veto) . (24) 
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FIG. 4: Differential cross sections versus the invariant mass of the top-quark and the heavy 
flavor (i.e. Mtc or Mtb) for the sequential Z model and the SM single top-quark backgrounds: 
qq' -^W* ^th and qg tbj at the LHC. 

In this way, the qg tbj is reduced to a level smaller than qq' — > tb. As we have explained 
above, we require to see only one heavy-flavor jet with the top-quark. Therefore, the subpro- 
cess bg — >• tjj is reduced to a negligible level. After imposing the cuts in Eqs. and (j^ . 
we show in Fig. |3] the differential cross sections versus the invariant mass of the top-quark 
and the heavy flavor (c in the signal and b in the background). In the figure, we illustrate the 
signal with the sequential Z model. The total background is still about a factor of 5 larger 
than the signal. We have to rely upon the secondary vertex mass method or D-, D*-tagging 
to further separate the charmed and the bottom jets. We are going to explain it in the next 
subsection. 
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D. Charm Tagging 



Heavy quark flavor tagging is in general quite successful up to some limitation. We briefly 
describe it here. With the silicon vertex detector, one can use the presence of a secondary 
vertex in a jet to identify it as a heavy- flavor jet. The presence of a secondary vertex in a 
silicon vertex detector is in general due to the long decays of a bottom or charmed hadron. 
Here one requires at least two tracks (the minimum to form a secondary vertex) to meet 
at a point far away enough from the interaction point. A positive tag is placed when the 
secondary vertex is more than two standard deviations from the interaction point. 

Once a jet is identified with as a heavy- flavor jet, one can measure the secondary vertex 
mass (the invariant mass of the hadrons at the secondary vertex) to further distinguish 
between the charmed and bottom jets. A distinctive figure shown in Ref. js^ clearly shows 
the difference among the charmed, bottom, and uds-iets. The bottom jet has the largest 
secondary vertex mass with a tail up to 4 GeV, while the charmed jet has a secondary 
vertex mass ranging from to 2 GeV with a peak around 1 GeV. The light quark jets have 
the smallest secondary vertex masses. One can make use of the Monte-Carlo templates to 
determine the fractions of charm, bottom, and other light quarks in a jet sample. 

Another method is to identify the D* and D mesons, which the prompt charm-quark 
hadronizes into. It has been used to measure the prompt charmed mesons production 
at the Tevatron js^. One can reconstruct the charmed mesons in the following decay 
modes: K-tt+, D*+ D°7r+ with K~n+, D+ K~n+7r+, 07r+ 

with — » K^K^, and their charge conjugates. Details of reconstructing these charmed 
mesons can be found in Ref. js^. The most important criterion is to distinguish between 
the prompt charmed mesons and those from bottom meson decays. These two sources can be 
separated using the impact parameter of the net momentum vector of the charm candidate 
to the beamline. Prompt charmed mesons will point back to the beamline because the 
charm-quark hadronizes immediately after it is produced. Therefore, one can have some 
success in tagging the prompt charmed meson together with a single top-quark. However, 
we anticipate the efficiency not to be too high. The realistic efficiency is beyond the scope 
of the present paper. 

We summarize our findings for the LHC as follows. 

1. The FCNC production of tc + tc is mainly via on-shell production of the Z' boson. The 
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most likely scenario is that the Z' boson is first discovered in the gold-plated channel, 
the Drell-Yan process. We then search for the production of a single top-quark and 
a charmed jet in the hadronic decay of the Z' boson. Since the single top-quark and 
the charmed jet originate from the Z' decay, we impose a very large cut to reduce 
the background. The production rate of tc -|- tc for Mz> = 1 TeV is of the order of 1 
fb for several typical Z' models that we study in this work. 

2. The most serious irreducible background is the SM single top-quark production as- 
sociated with a bottom-quark. The collider signatures for a charmed jet and for a 
bottom jet are similar. Both have a secondary vertex in the silicon vertex detector. 
One may be able to use the secondary vertex mass method or to use the D, D*-meson 
tagging to distinguish between the charmed and bottom jets. However, experimental 
separation of charmed and bottom jets is still uncertain, so one would expect some 
difficulty in getting a clean signal. One has to rely on an accurate estimation of the 
SM background in order to extract the signal. 



E. e+e" -^tc + tc at ILC 



At linear colliders such as the ILC, only the s-channel diagram contributes to the process 
e~^e~ tc or tc. The differential cross section can be adapted from the above formulas and 
it reads 

da 3(71/9 1 



dcosO 



GAtcs s\, 



- {\Q1? - \Qr?) (iQif - \Qt?) i-^cut + t^t,) 

+Am,mt (IQIP + |Q^p) 7^e {QtQf) s . 



(25) 



We show in Fig. |31the cross sections of tc + tc production for y/s = 0.5 to 1.5 TeV with a 
fixed Z' mass of 1 TeV. 

Unlike the case of the LHC, the detection of tc + ic at an e~^e~ collider is much more 
straight-forward because the SM single top-quark production proceeds through •j-t-q and 
Z-t-q FCNC couplings (g = u,c) that are GIM suppressed One can measure under 

the Z' peak the cross sections for tt and tc + ic, and thus determine the parameter x. In 
fact, the ILC js^ may have the option of tuning the center-of-mass energy of the collision. 
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FIG. 5: Production cross sections for e~^e~ tc + tc at a linear e~^e~ collider versus the center- 
of-mass energy. Results for the five models mentioned in the text are presented. 

Then one can tune it to the Z' mass to maximize the production cross section, as shown 
in Fig. With the sihcon vertex detector one can detect events with a heavy flavor (the 
charm-quark) and a single top-quark. We show in Fig. IHlthe ratio of a{tc + tc)/a{tt) versus 
the parameter x for the Z' models that we consider in this paper. For a reasonable range of 
X the ratio is about 10"*^ — 10~^. Moreover, the number of FCNC events under the Z' peak 
is large, of the order of 10'^ — 10^ events for an integrated luminosity of 100 fb^^. Therefore, 
we conclude that the ILC will be much better than the LHC in probing the FCNC process 
of tc + ic production via a Z' boson. 

V. CONCLUSION 

In the framework of models with an extra U{1) gauge boson that has family non-universal 
couplings, one can induce tree-level FCNC couplings in the up-type and/or down-type quark 
sector after diagonalizing their mass matrices. In the models considered in this paper, it is 
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FIG. 6: The ratio (j{tc + tc)/a{tt) versus x with y/s = Mz' = 1 TeV for the models of sequential 
Z, Zlr, Z^, Z^, and Z^. Note that the curves for Z^, Z^, and Z^ overlap because the ratios of 
the left-handed to the right-handed couplings are the same for these three Z' models. 

possible to have tree-level Z'-t-c and Z'-c-u couplings. We have studied the collider signature 
of associate top-charm production at both LHC and ILC and discussed the constraint of a 
tree-level Z'-c-u coupling from D — D mixing. 

For the LHC, the main contribution to the associate tc + ic production is from the s- 
channel diagram qq Z'* — tc + tc. The total cross section can be of the order of 1 fb 
for Mz' = 1 TeV in the framework of Z' models that we have discussed. The most serious 
irreducible background is the SM single top-quark production associated with a bottom- 
quark. It is found that the total SM background is larger than the signal. Therefore, in 
order to extract the FCNC signal of the Z' boson, a detailed Monte-Carlo simulation study 
of the SM background is required. 

At the ILC, the situation is more promising given the fact that the signal is almost 
background free. For ^/s ~ M^/, the cross section can reach a size of more than 100 fb. 
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Away from the resonance, there is still a region where the cross section can be larger than 
the threshold of observability 0.01 fb for such a clean process. 
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